Categories
Beautiful Soup

Web Scraping with Beautiful Soup — Parent and Sibling Elements

Spread the love

We can get data from web pages with Beautiful Soup.

It lets us parse the DOM and extract the data we want.

In this article, we’ll look at how to scrape HTML documents with Beautiful Soup.

Going Up

We can move up the tree with Beautiful Soup.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
title_tag = soup.title
print(title_tag)
print(title_tag.parent)

Then the first print call prints:

<title>The Dormouse's story</title>

And the 2nd print call prints:

<head><title>The Dormouse's story</title></head>

So we see the head tag with the parent property.

.parents

We can iterate over all element’s parents with the .parents property.

For example, we can write:

from bs4 import BeautifulSoup
html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
link = soup.a
print(link)
for parent in link.parents:
    print(parent.name)

We get the first a element with soup.a .

So the first print call is:

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

The 2nd print calls prints all the parents of the a element, so we get:

p
body
html
[document]

Going Sideways

We can get siblings elements with Beautiful Soup.

For example, we can write:

from bs4 import BeautifulSoup

sibling_soup = BeautifulSoup(
    "<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.prettify())

Then we get:

<a>
 <b>
  text1
 </b>
 <c>
  text2
 </c>
</a>

printed.

.next_sibling and .previous_sibling

We can get the next sibling with the .next_sibling property and the previous sibling with the .previous_sibling property.

For example, we can write:

from bs4 import BeautifulSoup

sibling_soup = BeautifulSoup(
    "<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.b.next_sibling)
print(sibling_soup.c.previous_sibling)

We see:

<c>text2</c>
<b>text1</b>

printed from the print calls.

The strings in the tags aren’t siblings since they don’t have the same parent.

.next_siblings and .previous_siblings

We can get multiple siblings with the .next_siblings and .previous_siblings properties.

For example, if we have:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.a.next_siblings:
    print(repr(sibling))

Then we see all the siblings next to the first a element:

u',n'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u' andn'
<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
u';nand they lived at the bottom of a well.'

printed.

We can do the same with the previous_siblings property:

from bs4 import BeautifulSoup

html_doc = """<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc, 'html.parser')
for sibling in soup.find(id="link3").previous_siblings:
    print(repr(sibling))

And we see:

u' andn'
<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
u',n'
<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
u'Once upon a time there were three little sisters; and their names weren'

printed.

Conclusion

We can get parent and sibling nodes with Beautiful Soup.

By John Au-Yeung

Web developer specializing in React, Vue, and front end development.

2 replies on “Web Scraping with Beautiful Soup — Parent and Sibling Elements”

Leave a Reply

Your email address will not be published. Required fields are marked *